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Description 

Field of The Invention 

[0001] The present invention is related to the field of 
digital image processing and more particularly to meth- 
ods and associated apparatus for reducing quantization 
artifacts in a hierarchical image storage and retrieval 
system that employs the quantization of residual image 
components. 

Background Of The Invention 

[0002] I mage pyramids provide an efficient means for 
the storage and retrieval of digital images at various res- 
olutions. It is customary in such hierarchical image stor- 
age and retrieval systems to construct a multiple of im- 
age resolutions by repeatedly filtering and subsampling 
a high resolution image using a scheme similar to the 
teachings found in the patent to Melnychuck and Jones 
(U. S. Pat. No. 4,969,204). In such cases, the higher 
resolution components are usually stored in the form of 
quantized residuals that are entropy encoded to save 
storage space. In the context of the present invention, 
we will make reference to one such system, namely the 
Kodak Photo CD system, as an example to demonstrate 
the fundamental concepts underlying the invention. 
However, it should be noted that this example is only 
used to give the reader an insight as to the operation of 
one or more embodiments of the invention with the un- 
derstanding that other hierarchical image storage and 
retrieval systems with other resolutions or arrange- 
ments that store the high resolution components as re- 
siduals may be chosen to suit specific needs without de- 
tracting from the teachings of the present invention. 
[0003] The Kodak Photo CD system is composed of 
an image hierarchy or pyramid in which the highest res- 
olution image contains 3072 x 2048 pixels and is re- 
ferred to as the 16BASE image. This resolution is in 
most cases adequate to produce photographic quality 
originals on an appropriate digital output device. The 
next level of hierarchy, referred to as the 4BASE, is com- 
posed of 1 536 x 1 024 pixels and is adequate to generate 
a high quality HDTV display, or a small-sized photo- 
graphic quality print on an appropriate digital output de- 
vice. The lower resolution levels are, respectively, the 
BASE image consisting of 768 x 51 2 pixels (suitable for 
NTSC/PAL/SECAM television viewing), the BASE/4 im- 
age consisting of 384 x 256 pixels (suitable for sub-NT- 
SC displays), and the BASE/16 consisting of 192 x 128 
pixels (suitable for displaying thumbnail images). The 
complete image hierarchy is constructed from the 
16BASE image using the teachings found in the #204 
patent to Melnychuck and Jones previously cited. The 
BASE/1 6, BASE/4, and the BASE images are stored on- 
to a digital storage medium such as a CD in uncom- 
pressed form while the 4BASE and 16BASE images are 
represented as residuals and are quantized and com- 



pressed by using entropy coding prior to their storage 
on the CD. 

[0004] A quantizer is a many-to-one mapping and, as 
such, the quantized data often contains degradations 

5 compared to the original unquantized data. The degra- 
dations can be particularly significant when the input da- 
ta has a distribution (histogram) containing long tails, as 
those values are clipped to the outermost quantization 
level. In a hierarchical imaging system such as the Ko- 

10 dak Photo CD system, the residual image usually has a 
symmetric distribution with a high peak and decaying 
tails, sometimes approximated by a Laplacian (two-sid- 
ed exponential) distribution. The minimum-mean- 
squared-error quantizer that accommodates such a dis- 
tribution contains dense inner level (in the region where 
the signal peaks) and sparse outer level (around the 
tails). An example of how a quantizer processes treats 
signal values is shown in the table of Fig. 10. It is based 
on the principal that by introducing a larger error in those 

20 values where the signal occurs infrequently and a small- 
er error at values where it occurs more frequently, the 
overall average error can be minimized. 
[0005] As described in the teachings of the previously 
cited patent to Melnychuck and Jones, the construction 

25 of a high- resolution image is accomplished by adding 
the quantized residual to the interpolated low-resolution 
component from the previous stage. Thus, the error re- 
sulting Irom the quantization of the residual is propagat- 
ed to the higher-resolution images. Although the resid- 

30 uals with large magnitudes occur infrequently, it should 
be noted that they often represent high-contrast edges 
or details in an image, and the introduction of a large 
quantization error can create an annoying visual artifact 
or loss of detail in the high-resolution image. US-A- 

35 5,020, 1 20 discloses a method and apparatus for reduc- 
ing quantization artifacts in a hierarchical image system 
of the type that decomposes a high resolution image into 
low resolution image components including low resolu- 
tion pixel value, and residual image pixel values. One of 

40 the objectives of the present invention is to introduce a 
method whereby the low-resolution image whose image 
quality is less critical can be modified in order to mini- 
mize the quantization errors in the high-resolution im- 
age. 

45 [0006] The aforementioned hierarchical storage tech- 
nique is combinable with the insert bn and removal of a 
digital watermark according to the teachings of EP-A- 
651,554 by Rabbani et al., to form a distributed image 
system. In a distributed image system it is common to 

50 deliver an image of compromised image quality for the 
purposes of browsing or proofing. The compromise can 
be made with the use of a removable watermark. Prima- 
ry use of such a system is by professional portrait pho- 
tographers who distribute a plurality of images to a cus- 

55 tomer for selection and approval. The watermark is a 
form of graphic overlay that may contain a copyright no- 
tice or information regarding the restricted use of the im- 
age. 
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[0007] Upon selection of the desired image by the 
customer, the professional delivers a high quality rendi- 
tion of the image either digitally or in hard copy form with- 
out the watermark. At all times the professbnal pos- 
sesses the sole means of generating and delivering un- 5 
marked high quality images. In a conventional photo- 
graphic system the means for generating would be the 
original negatives of the images; in a digital hierarchical 
system according to the invention of EP-A-651 ,554, the 
means for generating are higher resolution residual 10 
components. 

[0008] In a digital imaging system, and in particular 
one that includes a hierarchical form of digital storage 
and retrieval, the professional may use a suitable digital 
storage medium such as a CD for the distribution of *5 
proofs. In an unrestricted environment, the customer 
may choose a desired image resolution from the hierar- 
chy for the purposes of browsing, proofing, or hard copy 
fulfillment. In those instances where it is desirable for 
the professional to deliver the digital storage medium 20 
containing the entire image hierarchy to the customer, 
it is also most economical to record the entire image hi- 
erarchy once onto the digital storage medium and avoid 
having to make a second copy containing only low res- 
olution components for distribution. However, it is also 25 
desirable to restrict the use of selected high resolution 
components for the purpose of full image quality fulfill- 
ment until payment has been received. 
[0009] The invention of the Rabbani et al. application, 
previously referenced, teaches a method whereby the 30 
professbnal places a digital watermark on a selected 
image component. The removal of the watermark is 
done through an additional image component contain- 
ing the reverse of the watermark. The customer, having 
possession of the CD storage medium, would possess 35 
the means for generating his own high quality hard copy 
when authorized by the professional. 
[0010] When a watermark is applied to a low-resolu- 
tion image, the pixel values in that region of the low- 
resolution image affected by the watermark can become 40 
completely different from the pixel values in the high- 
resolution image corresponding to the same region. As 
a result, in those regions, the difference between the in- 
terpolated watermarked image and the high-resolution 
image can become fairly large which, given the nature 45 
of the quantizer, can result in large quantizatbn errors. 
These errors often appear as a visual trace of the wa- 
termark in the high-resolution image and significantly 
impair the visual quality of the high-resolution image. 
Minimization of such errors is addressed by the present so 
invention. 

[001 1] In accordance with the invention, the above ob- 
jects are met by the method and apparatus defined in 
claims 1 and 5. 

[0012] From the foregoing it can be seen that it is a 55 
preferred object of the present invention to provide a 
method and associated apparatus for improving the 
quality of a high-resolution image component by modi- 



fying a lower-resolution image component whose image 
quality is less critical. 

[0013] It is another object of the present invention to 
improve the quality of a high-resolution image compo- 
nent by modifying a lower-resolution image component 
whose image quality is less critical when a watermark 
is inserted and removed. 

[0014] Yet another object of the present invention is 
the reduction of quantizatbn artifacts in the addition and 
removal of a digital watermark to and from a selected 
resolution image where the watermark removal record 
is placed in a higher resolution image component. 
[001 5] The above and other objects of the present in- 
vention will become more apparent when taken in con- 
junction with the following description and drawings 
wherein like characters indicate like parts and which 
drawings form a part of the present invention. 

Brief Description Of The Drawings 

[0016] Figure 1 illustrates, in block diagram form, a 
PRIOR ART hierarchical image decomposition scheme 
combined with a scheme for inserting a digital water- 
mark in a low-resolution image component and placing 
a watermark removal component in a high-resolution re- 
sidual. 

[0017] Figure 2 illustrates, in block diagram form, a 
PRIOR ART hierarchical image reconstruction scheme 
for recovering a high-resolution image that has been de- 
composed by the scheme illustrated in Figure 1 . 
[001 8] Figure 3 is an example of the brightness com- 
ponent of a BASE image in pixel value form. 
[0019] Figure 4 is an example of a watermarked 
BASE image in pixel value form. 
[0020] Figure 5 is an example of the pixel values of a 
4BASE image corresponding to the pixel values of the 
BASE image of Figure 3 

[0021] Figure 6 illustrates the relative X.Y position of 
the pixels used in a linear interpolation scheme. 
[0022] Figure 7 illustrates the interpolated water- 
marked BASE image pixel values corresponding to the 
pixel values of the watermarked BASE image of Figure 
4. 

[0023] Figure 8 illustrates the pixel values of a resid- 
ual image formed as the difference between the images 
in Figures 5 and 7. 

[0024] Figure 9 illustrates the pixel values of the re- 
sidual image of Figure 8 after quantization. 
[0025] Figure 1 0 illustrates, in table form, the quantiz- 
er input-output characteristics used in generating the 
quantized residual image pixel values of Figure 9. 
[0026] Figure 11 illustrates the pixel values of the re- 
constructed 4BASE image using the PRIOR ART 
schemes of Figures 1 and 2. 

[0027] Figure 12 illustrates the improvement to the 
PRIOR ART system of Figure 1. 
[0028] Figure 13 illustrates, in a flow diagram form, 
the steps necessary for the adjustment of the values of 
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the low-resolution image pixels during the first pass of 
the present invention. 

[0029] Figure 14 illustrates the pixel values of a mod- 
ified watermarked BASE image resulting from a first 
pass of the present invention. 
[0030] Figure 15 illustrates the pixel values of a re- 
constructed 4BASE image using the pixel values of the 
modified watermarked BASE image of Figure 14. 
[0031] Figure 16 is a flow diagram of the procedure 
employed in STEP 1 of the second pass of the present 
invention to adjust the pixel values of the low-resolution 
image. 

[0032] Figure 17 is a flow diagram of the procedure 
employed in STEP 2 of the second pass of the present 
invention to adjust the pixel values of the low-resolution 
image. 

[0033] Figure 18 is a flow diagram of the procedure 
employed in STEP 3 of the second pass of the present 
invention to adjust the pixel values of the low-resolution 
image. 

[0034] Figure 1 9 illustrates the pixel values of a mod- 
ified watermarked BASE image resulting from a second 
pass of the present invention. 

[0035] Figure 20 illustrates the pixel values of a re- 
constructed 4BASE image using the watermarked 
BASE image of Figure 19. 

Detailed Description Of The Invention 

[0036] The present invention generally addresses the 
potential problem of artifact generation in a high resolu- 
tion image caused by the quantization of residuals. One 
example is when the high resolution image contains 
high contrast edges or areas of detail that can give rise 
to a large residual value. Consequently, the quantization 
or clipping of this residual value can degrade the quality 
of the high resolution image reconstructed from this re- 
sidual. 

[0037] Another example is when a watermark is in- 
serted into the low resolution image. In this case, the 
pixel values in that region of the low-resolution image 
affected by the watermark can become completely dif- 
ferent from the pixel values in the high-resolution image 
corresponding to the same region, thus resulting in large 
residual values. The error resulting from the quantiza- 
tion of such large residual values often appears as a vis- 
ual trace of the watermark in the high-resolution image. 
[0038] A PRIOR ART watermark system arrangement 
is illustrated in Figure 1 . This arrangement is the same 
as the arrangement illustrated in Figure 2 of US. Patent 
Application Serial No. 146,371, referenced above. In 
Box 34, a watermark W is first inserted in the BASE im- 
age I to arrive at a watermarked BASE image l w . This 
watermarked BASE image is then interpolated to the 
size of the 4BASE image in interpolator 24 using linear 
interpolation. A difference is formed between the origi- 
nal 4BASE image taken from the output of a decompo- 
sition apparatus 101 and the interpolated watermarked 



BASE image in subtracter 32 to form a modified 4BASE 
residual that serves as the watermark removal record. 
This modified 4BASE residual is encoded by encoder 
20, encrypted in encryption unit 26 and stored on a CD 
5 storage medium 10 along with the watermarked BASE 
image l w in unencrypted form in addition to the original 
unencrypted BASE/4 and BASE/16 images. Additional- 
ly, a 16 BASE residual is formed and added to the CD 
storage medium 10. Note that except for a negligible in- 
to crease in the size of the compressed 4BASE residual, 
there are no additional memory requirements since the 
watermark removal component is embedded in the 
4BASE residual. 

[0039] Throughout this description of the invention 

is reference is made to the forming, merging, or differenc- 
ing of images. It is understood by those skilled in the art 
that the pixel values representing images are manipu- 
lated, not the images themselves. 
[0040] For browsing or proofing, the PRIOR ART sys- 

20 tern of Figure 2 is employed. The PRIOR ART system 
arrangement of Figure 2 is substantially the same as the 
arrangement illustrated in Figure 3 of US, Patent Appli- 
cation Serial No. 146,371, referenced above. The user 
retrieves the BASE/16, BASE/4, or watermarked BASE 

25 images directly without decryption from the CD storage 
medium 10. Upon authorization, the user inputs a de- 
cryption key to the data decryption unit 28 to decrypt the 
modified 4BASE residual. The watermarked BASE im- 
age is decoded in decoder 30 and interpolated using lin- 

30 ear interpolation and is added to the decrypted modified 
4BASE residual in the reconstruction apparatus 201 to 
recover the 4BASE image. Details of the reconstruction 
apparatus 201 can be obtained from Figure 2, and the 
associated description found in US. Patent No. 

35 4,969,204. In general, if the modified 4BASE residual is 
quantized prior to its storage on the CD 10 (which is usu- 
ally the case), the exact 4BASE image cannot be recov- 
ered. The discrepancy between the original 4BASE im- 
age and the 4BASE image recovered according to the 

40 above procedure depends on the coarseness of the 
quantizer in addition to the nature of the watermark. The 
quantizers used in the PRIOR ART systems of Figures 
1 and 2 are part of the encoders 20 
[0041] The description of the problem encountered 

45 with the PRIOR ART systems of Figures 1 and 2 can be 
clarified by using an example. Assuming that an image 
is represented by a two-dimensional array of pixels with 
the pixels being represented by numbers with indices (i, 
j) denoting the location of a pixel at the ith row and the 

50 jth column, where i=l,...,N,, and j=l N p , and N t and N p 

are, respectively, the number of the lines and the 
number of pixels per line in the image, and that the 
brightness of each pixel is represented by an 8-bit 
number, i.e., a number in the range of 0 to 255. Now 

55 consider an example where, for the purpose of simplic- 
ity, a taw-resolution image, hereafter referred to as the 
BASE image, is composed of a 5x5 array of pixels as 
illustrated in Figure 3. Referring to Box 34 in Figure 1, 
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a watermark W is added to this image to create the wa- 
termarked BASE image of Figure 4. The continuous line 
in Figure 4 shows the region affected by the watermark. 
In this example, the effect of the watermark is to replace 
the value of the pixels overlaid by the watermark W by 
a uniform brightness value of 200 so as to create a high 
contrast between the watermark area and the remaining 
pixels in the BASE image. 

[0042] Figure 5, hereafter referred to as the 4BASE 
image, shows an image that consists of a 9x9 array of 
pixels corresponding to the high resolution rendition of 
Figure 3. Referring to Box 24 in Figure 1, the water- 
marked BASE image in Figure 4 is interpolated to create 
a prediction for the 4BASE image. The interpolation 
scheme used in this example is linear interpolation and 
is described with reference to Figure 6. For best system 
performance the interpolator 24 should match the inter- 
polator 49 (shown in Figures 16, 17, and 18). The four 
pixels in the low-resolution image labeled X 1 through X4 
are linearly interpolated to create the pixels Y A through 
Y 3 according to the following relationships: 

Y 1 = (X 1+ X 2 V2, 



Y 2 = (X 1+ X 3 )/2, 



Y 3 = (x 1+ x 2+ x 3+ x 4 y4, 

where, in order to insure that the Y { values are also 8-bit 
integers, all fractions are rounded to the nearest integer 
and a fractional value of 0.5 is truncated down, e.g., a 
value of 58.50 is truncated to the integer value of 58. 
Figure 7 illustrates the interpolated watermarked BASE 
image according to the above interpolation scheme 
where the boxes highlight the pixels that correspond to 
the tow-resolution image (watermarked BASE image) of 
Figure 4. Referring to the subtracter 32, in Figure 1 , the 
modified 4BASE residual image is formed as the pixel- 
by-pixel difference of the 4BASE image of Figure 5 and 
the interpolated watermarked BASE image of Figure 7. 
Figure 8 illustrates this residual image which is the input 
to Box 20 in Figure 1 . It can be easily seen that if the 
residual image of Figure 8 is stored on the CD storage 
medium 1 0 without any quantization, the original 4BASE 
image can be completely recovered. However, in most 
applications, to save storage space, the residual image 
is quantized prior to its storage. 
[0043] An example of the characteristics of a quantiz- 
er used in conjunction with the Kodak Photo CD system 
is shown in Figure 10. For this example, if the input to 
the quantizer is any number in the range of -47 to -255, 
it will be quantized (clipped) to the value of -48. Using 
the quantizer in Figure 1 0 to quantize the residual image 
of Figure B results in the quantized residual shown in 
Figure 9 which is then entropy encoded (e.g., Huffman 



encoded) and stored on the CD storage medium 10. As 
shown in Figure 2, the 4BASE image is recovered by 
adding the quantized residual of Figure 9 to the interpo- 
lated watermarked BASE image of Figure 7. This results 

s in the reconstructed 4BASE image shown in Figure 11. 
Due to the errors introduced by the quantizer, the 
4BASE image reconstructed in this way is significantly 
different from the original 4BASE image especially in the 
regions affected by the watermark. The underlined pix- 

10 els in Figure 11 are those that substantially differ from 
the original 4BASE image of Figure 5. The standard de- 
viation of the pixel-by-pixel error between the original 
4BASE image and the reconstructed 4BASE image, al- 
so referred to as the root-mean-squared-error (RMSE), 

15 is computed to be 63.89. A large portion of this error 
results from the fact that the quantizer clips the value of 
any negative residual with a magnitude larger than 48. 
If this reconstructed 4BASE image is printed with a hard 
copy device or viewed on a monitor, it would display a 

20 visible trace of the watermark. 

[0044] Referring to Figure 12, in accordance with the 
teachings of the present invention, the watermarked 
low-resolution image l w at the output of Box 34 is mod- 
ified in an image modifying unit 36 prior to its interpola- 

25 tion and storage on the CD storage medium in such a 
way as to reduce the artifacts that result from the quan- 
tization of the modified residual in the encoder unit 20. 
The extent of the modification depends on the charac- 
teristics of the quantizer and the watermark. For exam- 

30 pie, for a fine quantizer which contains many quantiza- 
tion levels, the changes in the visual quality of the wa- 
termarked BASE image, as a result of the present in- 
vention, will be negligible. On the other hand, if a coarse 
quantizer with a high degree of clipping is used along 

35 with a high-contrast watermark, the changes will be 
more perceptible and will generally appear as a reduc- 
tion in the watermark contrast. However, it should be 
noted that in almost all applications of interest, the ben- 
efits of recovering a high quality high-resolution image 

40 far outweigh any disadvantages of reducing the water- 
mark contrast. 

[0045] The present invention introduces a method 
whereby the pixel values in a low-resolution image are 
modified prior to their encoding and storage in order to 

45 reduce the quantization artifacts in the higher-resolution 
image. The objective is to introduce the smallest possi- 
ble magnitude changes in the low-resolution image pixel 
values that will also minimize the quantization error in 
the residual image while maintaining the viewability of 

50 the low-resolution image. In this way, the fidelity of the 
reconstructed high-resolution image can be improved 
while minimizing any compromises to the image quality 
of the low-resolution image. In one embodiment of the 
present invention, this technique can be used to improve 

55 the reconstruction quality of a high-resolution image in 
a hierarchical image storage and retrieval system at the 
expense of introducing minor changes in the low-reso- 
lution image. This is particularly useful for those appli- 
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cations where the image quality of a higher resolution 
image component is more critical than the image quality 
of the lower resolution image component. 
[0046] Referring to the flow diagram of Figure 13, 
which illustrates the functions performed by the image 
modifying unit 36, the difference (residual) between the 
value of each pixel in the low-resolution image and the 
value of its corresponding pixel in the high-resolution im- 
age is found In the Photo CD example, where a high- 
resolution image has twice as many pixels in each di- 
mension as the low-resolution image, the high-resolu- 
tion pixel location corresponding to the low-resolution 
pixel location (i.j) is (2i-1 ,2j-l) and the difference is found 
as the output of subtraction operation performed at cir- 
cle 50. The residual value (difference) from circle 50 is 
then passed through quantizer 52 to output a quantized 
residual value. The functional characteristics of a quan- 
tizer are shown in the chart of Figure 10. This value is 
then compared, in circle 60, against the residual value 
from circle 50 to determine if an error exists. If it does 
the low-resolution pixel value is adjusted at function 58, 
to make the quantization error zero. If no error exists no 
adjustment is required as stated in block 56. Each low 
resolution pixel value from 56 or 56 is stored in a storage 
buffer 70 pending the adjustment of all pixels forming 
the image. If a second pass is to be implemented it uses 
the pixel values stored in storage buffer 70. If a second 
pass is not to be implemented the pixels from storage 
buffer 70 may be output to the CD 10 and to the inter- 
polator 24. 

[0047] Carrying further the example discussed previ- 
ously, the residuals between the pixels in the water- 
marked BASE image of Figure 4 and their correspond- 
ing pixels in the 4BASE image of Figure 5 are found and 
adjusted according to the above scheme. The result af- 
ter a first pass is shown in Figure 14. For example, the 
result of subtracting the pixel value in location (1,1) of 
Figure 4 from its corresponding pixel value in location 
(1,1) of Figure 5 is +1. Referring to the quantizer char- 
acteristics of Figure 10, a residual value of +1 is quan- 
tized to the value +1 resulting in no error and thus no 
adjustment for that pixel. As another example, the result 
of subtracting the pixel value in location (4,3) of Figure 
4 from its corresponding pixel value in location (7,5) of 
Figure 5 is -1 32 which is quantized to the value -48. The 
smallest magnitude adjustment to the low-resolution 
pixel value that will reduce the quantization error to zero 
is -84. Thus the pixel value in location (4,3) of Figure 4 
is changed from 200 to 116. As is seen from Figure 14 
the effect of this process is to reduce the watermark con- 
trast in the watermarked BASE image. However, the 
quality of the reconstructed high-resolution image is sig- 
nificantly improved. 

[0048] Figure 1 5 illustrates the reconstructed 4BASE 
image corresponding to the modified watermarked 
BASE image of Figure 1 4. Again, the underlined pixels 
differ from the original 4BASE image as a result of the 
quantization error. The RMSE between the original 



4BASE image and the reconstructed 4BASE image of 
Figure 15 is 3.09. This constitutes a significant improve- 
ment compared to the RMSE of 63.89 corresponding to 
the reconstructed 4BASE image of Figure 11 . As is seen 

s from Figure 1 5, a large portion of this error is due to the 
three pixels in locations (2,7), (2,8), and (2,9), which 
have been highlighted in a box. In the present invention, 
in order to further reduce the quantization error, the low 
resolution pixel values are adjusted in a second pass 

10 which consists of three steps as outlined below. In de- 
scribing the various steps of the second pass, reference 
is made to Figure 6 which illustrates a neighborhood 
containing the four low-resolution pixels labeled X, 
through X4 and the resulting linearly interpolated pixels 

is Y, through Y 3 . 

[0049] Starting from the top-left pixel of the low-reso- 
lution image resulting from the first pass stored in stor- 
age buffer 70, and scanning the image from left to right 
and from top to bottom in a raster fashion, STEP(S) 1, 

20 2 and 3 described below are applied in succession to 
every pixel in the low-resolution image with the latest 
adjustment overwriting the previous pixel value in the 
storage buffer 70: 

[0050] STEP 1 : For a given low-resolution pixel value 

25 x v the interpolated pixel value Y 1 is computed accord- 
ing to the interpolation equation and its corresponding 
residual value is found. The value of the low-resolution 
pixel X2 (to the right of X A ) is then adjusted by the small- 
est magnitude needed to eliminate the quantization er- 

30 ror of that residual. Note that as a result of this adjust- 
ment, the residual may sometimes be quantized to a val- 
ue different from what it was quantized to prior to the 
adjustment. This is also true for STEP(S) 2 and 3 of the 
algorithm. A flow diagram describing the procedure em- 

35 ployed in STEP 1 is depicted in Figure 16. 

[0051] STEP 2: For the same tow-resolution pixel val- 
ue X 1 as in STEP 1 , the interpolated pixel value Y 2 is 
computed according to the interpolation equation and 
its corresponding residual value is found. The value of 

40 the low resolution pixel X 3 (below is then adjusted 
by the smallest magnitude needed to eliminate the 
quantization error of that residual. A flow diagram de- 
scribing the procedure employed in STEP 2 is depicted 
in Figure 17. 

45 [0052] STEP 3: For the same low-resolution pixel val- 
ue X 1 as in STEP 1 , the interpolated pixel value Y 3 is 
computed according to the interpolation equation and 
its corresponding residual value is found. The value of 
the low resolution pixel X4 (to the right and below of X.,) 

50 is then adjusted by the smallest magnitude needed to 
eliminate the quantization error of that residual. A flow 
diagram describing the procedure employed in STEP 3 
is depicted in Figure 16. 

[0053] It should be noted that at any step of the sec- 
55 ond pass, the calculations are based on the modified 
tow-resolution pixel values that reflect all the adjust- 
ments up to that stage. Also, each pixel in the low-res- 
olution image may be adjusted up to three times during 
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the second pass. For those pixels that are located on 
the image borders, only a part of the neighborhood as 
defined in Figure 6 might exist. In such cases, only those 
steps for which a neighborhood is defined are carried 
out. The modified low-resolution image after the com- 
pletion of the second pass would usually generate a very 
high-quality reconstruction of the high -resolution image. 
[0054] The application of the second pass of the 
present invention to the numerical example discussed 
before will help to illustrate further the operation of each 
step. Scanning the image of Figure 14 in a raster fash- 
ion, it is found that no adjustment is necessary until the 
pixel location (1 .3) is reached. Denoting the value of this 
pixel by X-,, the value of the interpolated pixel Y A inSTEP 
1 is found to be 92. The corresponding high -resolution 
pixel in Figure 5 is located at (1,6) and has a value of 
70. The residual is thus -22 which results in no quanti- 
zation error based on the quantizer characteristics of 
Figure 10. Similarly, the residual, corresponding to the 
interpolated value Y 2 in STEP 2, is +1 and results in no 
quantization error. However, the value of Y 3 in STEP 3 
is found to be 92, which, once subtracted from the value 
75 located at position (2,6) in Figure 5, results in a re- 
sidual of -17. This residual is quantized to a value of -16, 
introducing a quantization error of -1 . The smallest mag- 
nitude adjustment needed to eliminate this quantization 
error will adjust the pixel value X 3 , located at (2,4) in 
Figure 1 4, to 1 1 9 instead of its current value of 1 1 8. After 
the adjustment, the residual value becomes -16 which 
is quantized without error. This procedure is continued 
until all the low resolution pixel values are processed. In 
this example, the low-resolution pixels that are located 
at (3,5), (4,2), (4,5), and (5,5), are adjusted three times 
during the second pass. 

[0055] Proceeding with the example, the completion 
of the second pass results in the modified watermarked 
BASE image of Figure 1 9. Note that while most pixel 
values in Figure 19 have undergone minor adjustments 
compared to their corresponding values in Figure 14, 
the two pixels in locations (2,4) and (2,5) have signifi- 
cantly changed. This was necessary to reduce the large 
quantization error on the three pixels in the image of Fig- 
ure 15 that are enclosed in a box. Figure 20 shows the 
reconstructed 4BASE image corresponding to the mod- 
ified watermarked BASE image of Figure 19. Again, the 
underlined pixels differ from the original 4BASE image 
as a result of the quantization error. It can be seen that 
all the pixel values in Figure 20 are within one unit or 
less of their corresponding value in the original 4BASE 
image of Figure 5 except for the pixel located at (9,9), 
which shows a difference of 2 units. The RMSE between 
the original 4BASE image and the reconstructed 4BASE 
image of Figure 20 is only 0.54, indicating the significant 
improvement resulting from the second pass. 
[0056] The high-resolution image, resulting from the 
present invention (Figure 20), compared with the PRI- 
OR ART image of Figure 1 1 , represents a significant im- 
provement. The price paid for this significant improve- 



ment is the replacement of the watermarked BASE im- 
age of Figure 4 with the modified watermarked BASE 
image of Figure 1 9 which has a reduced watermark con- 
trast. In most practical applications, the slight compro- 

s mise in the usefulness of the low-resolution image com- 
ponent is far outweighed by the image quality improve- 
ments in the high-resolution image. 
[0057] The two-pass algorithm described above con- 
stitutes a preferred method for achieving the objectives 

10 of the present invention, namely, finding the smallest 
possible magnitude changes in the low-resolution im- 
age pixel values that also aim at minimizing the quanti- 
zation error in the residual image. It should be noted that 
several variations to this main theme are possible. In 

15 general, based on simulations with the Photo CD image 
storage and retrieval system, it was found that the pre- 
ferred method resulted in a slightly better high-resolu- 
tion image quality (for the same degree of modification 
in the bw-resolution image) compared to the other var- 

20 iations. However, in some applications, other system 
considerations might make the use of the alternative ap- 
proaches more attractive. In what follows, a few of these 
variations are briefly described with reference to the pre- 
ferred method. 

25 [0058] In one variation, only the first pass of the algo- 
rithm, as described above, is implemented to create a 
reasonably high-quality, high-resolution image and the 
second pass is omitted to reduce the computational 
complexity. In another variation, the first and the second 

30 passes are combined into a single pass in the following 
way. Starting from the top-left pixel of the low-resolution 
image, the value of the pixel is first modified accord- 
ing to the teachings in the first pass of the preferred 
method. Next, the values of the pixels ^ X 3 , and X4 

35 are modified according to the STEP(S) 1 -3 of the second 
pass. Scanning the image from left to right and from top 
to bottom, this procedure is repeated for every pixel in 
the low-resolution image. The value of each pixel can 
be modified up to four times according to this procedure. 

40 However, since the order in which the adjustment takes 
place is different from the preferred method, in general, 
a different image will result. In yet another variation, the 
first pass is implemented as described in the preferred 
method, but the three steps in the second pass are irrv 

45 plemented as three separate passes. That is, starting 
from the top-left pixel of the low-resolution image, the 
image is scanned from left to right and top to bottom and 
the STEP 1 of the second pass is applied to all the pixels. 
Then, the same procedure is followed for STEP 2. Fi- 

50 nally, STEP 3 is implemented in a similar fashion. Again, 
this results in a different order in which the various pixels 
are processed. In yet another variation, after the execu- 
tion of the first pass as described in the preferred meth- 
od, the three steps in the second pass are carried out, 

55 but at each step, instead of adjusting the values of the 
pixels X2, X3, or X*. the value of the pixel X, is modified. 
Again, this will result in a different order of modifying the 
low-resolution pixel values than any of the schemes de- 
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scribed before. 

[0059] Furthermore, although the invention has been 
described with a watermark insertion unit which intro- 
duces high contrast edges, the man skilled in the art 
would realize that, as already mentioned earlier, for 
some images, the use of the invention provides an im- 
provement over the prior art even when no watermark 
insertion unit is provided. 



Claims 

1. A method for reducing quantization artifacts in a hi- 
erarchical image system of the type that decompos- 
es (101) a high resolution image into low resolution 
image components, including low resolution image 
pixel values, and residual image pixel values char- 
acterized in that the method further includes the 
steps of: 

a) determining (50) the difference values be- 
tween the pixel values in the low resolution im- 
age and the corresponding pixel values in the 
higher resolution image; 

b) comparing (60) each of the difference values 
against quantizer conversion values to deter- 
mine if an error would exist for an executed con- 
version; 

c) adjusting (58) the image pixel values within 
the low resolution image component if an error 
would exist, to a value that eliminates the error 
to form an adjusted low resolution image; and 

d) using the unadjusted image pixel values as 
part of the adjusted low resolution image com- 
ponent if an error does not exist. 

2. A method according to Claim 1 further comprising : 



component if an error does not exist. 

3. A method for according to Claim 2, wherein a step 
(22) of applying watermark pixel values to pixel po- 

5 sitions within the low resolution image is performed 
before step a) and said modified low resolution im- 
age is used as low resolution image. 

4. A method according to Claim 3, wherein the predict- 
10 ed higher resolution image of step e) is formed by 

sequentially interpolating (49) the pixel values of a 
first block of low resolution image pixel values and 
steps f) through i) are repeated for each pixel in the 
higher resolution image. 

75 

5. Apparatus for reducing quantization artifacts in a hi- 
erarchical image system of the type including 
means (101 ) that decomposes a high resolution im- 
age into low resolution components, including low 
resolutbn image pixel values, and residual image 
pixel values, characterized in that the apparatus fur- 
ther includes: 

means (50) for determining the difference val- 
ues between the pixel values in the low resolu- 
tion image and the corresponding pixel values 
in the higher resolution image; 
means (60) for comparing each of the differ- 
ence values against quantizer conversion val- 
ues to determine if an error would exist for an 
executed conversion; and 
means (58) for adjusting the image pixel values 
within the low resolution image component if an 
error would exist, to a value that eliminates the 
error to form an adjusted low resolution image 
component and using (56) the image pixel val- 
ue as part of the adjusted low resolution image 
component if an error does not exist 

6. Apparatus according to Claim 5 further comprising : 

means (49) for forming a predicted higher res- 
olution image by interpolating the pixel values 
of the adjusted low resolution image compo- 
nent; 

means (50) for determining the difference val- 
ues between the interpolated pixel values in the 
predicted higher resolution image and the cor- 
responding pixel values in the higher resolution 
image; 

means (60) for comparing each of the differ- 
ence values against quantizer conversion val- 
ues to determine if an error would exist for an 
executed conversion; 

means (58) for adjusting the image pixel values 
within the adjusted low resolution image com- 
ponent if an error would exist, to a value that 
eliminates the error to form a twice adjusted low 



e) forming a predicted higher resolution image 
by interpolating (49) the pixel values of the ad- 40 
justed low resolution image components of 
steps c) and d); 

f) determining (50) the difference values be- 
tween the interpolated pixel values in the pre- 
dicted higher resolution image and the corre- 45 
sponding pixel values in the higher resolution 
image; 

g) comparing (60) each of the difference values 
against quantizer conversion values to deter- 
mine if an error would exist for an executed con- so 
version; 

h) adjusting (58) further the image pixel values 
within the adjusted low resolutbn image com- 
ponent if an error would exist, to a value that 
eliminates the error to form an adjusted low res- 55 
olution image; and 

i) using (56) the unadjusted image pixel value 
as part of the adjusted low resolution image 
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resolution image component; and 
means (56) for using the image pixel value as 
part of the twice adjusted tow resolution image 
component if an error does not exist. 

7. Apparatus according to Claim 6, further comprising: 

means for applying pixel values to pixel posi- 
tions within the low resolution image to form a low 
resolution image component. 

8. A storage device (10) having stored thereon the ad- 
justed low resolution image components and the re- 
sidual image pixel values of any of Claims 5 to 7. 



Patentan sprue he 

1. Verfahren zum Reduzieren von Quantisierungsfeh- 
lern in einem hierarchischen Bildsystem, welches 
ein Bild hoher Auflosung in Bildkomponenten nied- 
riger Auflosung, bestehend aus niedrigaufgelosten 
Bildpixelwerten und verbleibenden Bildpixelwerten 
zerlegt, 

gekennzeichnet durch folgende Schritte: 

a) Bestimmen (50) der Differenzwerte zwi- 
schen den Pixelwerten im niedrigaufgelosten 
Bild und den entsprechenden Pixelwerten im 
hoheraufgelosten Bild; 

b) Vergleichen (60) jedes Differenzwerts mit 
Quantisierer-Umwandlungswerten, urn zu be- 
stimmen, ob bei einer durchgefuhrten Um- 
wandlung ein Fehler vorliegt; 

c) bei Vorliegen eines Fehlers, Angleichen (58) 
der Bildpixelwerte innerhalb der niedrigaufge- 
losten Bildkomponente an einen Wert, der den 
Fehler eliminiert, urn ein korrigiertes, niedrig- 
aufgelostes Bild zu erzeugen; und 

d) Verwenden der unveranderten Bildpixelwer- 
te als Teil der korrigierten, niedrigaufgelosten 
Bildkomponente, wenn kein Fehler vorliegt. 

2. Verfahren nach Anspruch 1 , gekennzeichnet durch 
folgende zusatzlichen Schritte: 

e) Erzeugen eines vorausberechneten Bildes 
hoherer Auflosung durch Interpolieren (49) der 
Pixelwerte der korrigierten, niedrigaufgelosten 
Bildkomponenten aus Schritt c) und d); 

f) Bestimmen der Differenzwerte zwischen den 
interpolierten Pixelwerten in dem vorausbe- 
rechneten Bild hoherer Auflosung und den ent- 
sprechenden Pixelwerten in dem Bild hoherer 
Auflosung; 

g) Vergleichen (60) jedes Differenzwerts mit 
Quantisierer-Umwandlungswerten, urn zu be- 
stimmen, ob bei einer durchgefuhrten Um- 
wandlung ein Fehler vorliegt; 



h) bei Vorliegen eines Fehlers, weiteres Anglei- 
chen (58) der Bildpixelwerte innerhalb der nied- 
rigaufgelosten Bildkomponente an einen Wert, 
der den Fehler eliminiert, um ein korrigiertes, 

s niedrigaufgelostes Bild zu erzeugen; 

i) Verwenden (56) des unveranderten Bildpixel- 
werts als Teil der korrigierten, niedrigaufgelo- 
sten Bildkomponente, wenn kein Fehler vor- 
liegt. 

10 

3. Verfahren nach Anspruch 2, dadurch gekennzeich- 
net, daB vor dem Schritt a) ein Schritt (22) durch- 
gefuhrt wird, in dem Pixelpositionen innerhalb des 
niedrigaufgelosten Bildes mit Wasserzeichen-Pi- 

15 xelwerten modifiziert werden und das modifizierte, 
niedrigaufgeioste Bild als niedrigaufgelostes Bild 
verwendet wird. 

4. Verfahren nach Anspruch 3, dadurch gekennzeich- 
20 net, da 3 das vorausberechnete, hoheraufgeloste 

Bild aus Schritt e) durch sequent ielles Interpolieren 
(49) der Pixelwerte eines ersten Blockes von Pixel- 
werten eines niedrigaufgelosten Bildes erzeugt 
wird und fur jedes Pixel im hoheraufgelosten Bild 
25 die Schritte f) bis t) wiederholt werden. 

5. Vorrichtung zum Reduzieren von Quantisierungs- 
fehlern in einem hierarchischen Bildsystem, in wel- 
chem eine Einrichtung (101) ein hochaufgelostes 

30 Bild in niedrigaufgeioste Bildkomponenten, beste- 
hend aus niedrigaufgelosten Bildpixelwerten und 
verbleibenden Bildpixelwerten zerlegt, 
dadurch gekennzeichnet, daB die Vorrichtung fol- 
gende zusatzliche Mittei aufweist: 

35 

ein Mittei (50) zum Bestimmen der Differenz- 
werte zwischen den Pixelwerten im niedrigauf- 
gelosten Bild und den entsprechenden Pixel- 
werten im hoheraufgelosten Bild; 

40 - ein Mittei (60) zum Vergleichen jedes Differenz- 
werts mit Quantisierer-Umwandlungswerten, 
um zu bestimmen, ob bei einer durchgefuhrten 
Umwandlung ein Fehler vorliegt; und 
ein Mittei (58) zum Angleichen der Bildpixel- 

45 werte innerhalb der niedrigaufgelosten Bild- 

komponente bei Vorliegen eines Fehlers an ei- 
nen Wert, der den Fehler eliminiert, um eine 
korrigierte, niedrigaufgeioste Bildkomponente 
zu erzeugen, und Verwenden (56) des unver- 

50 anderten Bildpixelwerts als Teil der korrigier- 

ten, niedrigaufgelosten Bildkomponente, wenn 
kein Fehler vorliegt 

6. Vorrichtung nach Anspruch 5, gekennzeichnet 
55 durch folgende zusatzliche Mittei: 

ein Mittei (49) zum Erzeugen eines vorausbe- 
rechneten Bildes hoherer Auflosung durch In- 
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terpolieren der Pixelwerte der korrigierten, 
niedrigaufgelosten Bitdkomponente; 
- ein Mittel (50) zum Bestimmen der Differenz- 
werte zwischen den interpolierten Pixelwerten 
in dem vorausberechneten Bild hoherer Auflo- 5 
sung und den entsprechenden Pixelwerten in 
dem Bild hoherer Auflosung; 

ein Mittel (60) zum Vergleichen jedes Differenz- 2. 
werts mit Quantisierer-Umwandlungswerten, 
um zu bestimmen, ob bei einer durchgef uhrten io 
Umwandlung ein Fehler vorliegt; 
ein Mittel (58) zum Angleichen der Bildpixel- 
werte innerhalb der korrigierten, niedrigaufge- 
losten Bildkomponente bei Vorliegen eines 
Fehlers an einen Wert, der den Fehler elimi- »s 
niert. um eine zweifach korrigierte, niedrigauf- 
geloste Bildkomponente zu erzeugen; und 
ein Mittel (56) zum Verwenden des unverander- 
ten Bildpixelwerts als Teil der zweifach korri- 
gierte, niedrigaufgelosten Bildkomponente, 20 
wenn kein Fehler vorliegt. 

7. Vorrichtung nach Anspruch 6, gekennzeichnet 
durch folgendes zusatzliches Mittel: 

25 

ein Mittel zum Anwenden von Pixelwerten auf 
Pixelpositionen innerhalb des niedrigaufgelo- 
sten Bildes, um eine niedrigaufgeloste Bild- 
komponente zu erzeugen. 

30 

8. Eine Speichereinrichtung (10), in der die angegli- 
chenen, niedrigaufgelosten Bildkomponenten und 
die verbleibenden Bildpixelwerte eines der Anspru- 

che 5 bis 7 gespeichert sind. 3. 

35 

Revendications 

1 . Procede pour require les artefacts de quantification 
dans un systeme imageur hierarchique du type qui <o 
decompose (101 ) une image a haute resolution en 4. 
composantes d'image a basse resolution, incluant 
les valeurs de pixels d'image a basse resolution et 
les valeurs de pixels d'image residuelles, caracteri- 
se en ce que le procede comprend en outre les eta- 45 
pes consistant a : 

a) determiner (50) les valeurs de difference en- 

tre les valeurs de pixels dans i'image a basse 5. 
resolution et les valeurs de pixels correspon- so 
dantes dans I'image a plus haute resolution ; 

b) comparer (60) chacune des valeurs de diffe- 
rence avec les valeurs de conversion de quan- 
tificateur pour determiner si une erreur existe 
pour une conversion executee ; 55 

c) ajuster (58) les valeurs de pixels cf image 
dans la composante d'image a basse resolu- 
tion si une erreur existe, a une valeur qui elimi- 



ne I'erreur afin de former une image a basse 
resolution ajustee ; et 

d) utiliser ies valeurs de pixels de I'image non 
ajustee comme partie de la composante d'ima- 
ge a basse resolution ajustee si une erreur 
n'existe pas. 

Precede* selon la revendication 1 comprenant en 
outre les Stapes consistant a : 

e) former une image predite par interpolation 
(49) a plus haute resolution des valeurs de 
pixels des composantes d'image ajustee a bas- 
se resolution des Stapes c) et d) ; 

f) determiner (50) les valeurs de difference en- 
tre les valeurs de pixels interpolees dans I'ima- 
ge predite a plus haute resolution et les valeurs 
de pixels correspondantes dans I'image a plus 
haute resolution ; 

g) comparer (60) chacune des valeurs de diffe- 
rence avec les valeurs de conversion du quan- 
tificateur pour determiner si une erreur existe 
pour une conversion executee ; 

h) ajuster (58) en outre les valeurs de pixels 
d'image dans la composante d'image ajustee a 
basse resolution si une erreur existe, a une va- 
leur qui eiimine I'erreur afin de former une ima- 
ge ajustee a basse resolution ; et 

i) utiliser (56) la valeur de pixel d'image non 
ajustee comme partie de la composante d'ima- 
ge ajustee a basse resolution si una erreur 
n'existe pas. 

Procede selon la revendication 2, dans lequel une 
etape (22) d'application de valeurs de pixels en fili- 
grane aux positions de pixels dans i'image a basse 
resolution est executee avant retape a) et ladite 
image modifiee a basse resolution est utilises com- 
me image a basse resolution. 

Procede selon la revendication 3, dans lequel I'ima- 
ge predite de retape e) a plus haute resolution est 
form6e en tnterpolant en sequence (49) les valeurs 
de pixels d'un premier bloc de valeurs de pixels 
d'image a basse resolution et les etapes f ) a i) sont 
r6petees pour chaque pixel dans I'image a plus hau- 
te resolution. 

Appareil pour reduire les artefacts de quantification 
dans un systeme imageur hierarchique du type in- 
cluant un moyen (101) qui decompose une image 
a haute resolution en composantes a basse reso- 
lution, incluant ies valeurs de pixels d'image a bas- 
se resolution et les valeurs de pixels d'image resi- 
duelles, caracterise en ce que Pappareil comprend 
en outre : 

un moyen (50) pour determiner les valeurs de 
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difference entre !es valeurs de pixels dans 
('image a basse resolution et les valeurs de 
pixels correspondantes dans Timage a plus 
haute resolution ; 

un moyen (60) pour comparer chacune des va- s 
leurs de difference aux valeurs de conversion 
de quantificateur pour determiner si une erreur 
existe pour une conversion executee ; et 
un moyen (58) pour ajuster les valeurs de pixels 
d'image dans la composante d'image a basse 10 
resolution si une erreur existe, a une valeurqui 
elimine I'erreur afin de former une composante 
d'image ajustee a basse resolution et utiliser 
(56) la valeur de pixel d'image comme partie de 
la composante d'image ajustee a basse n§so- w 
lution si une erreur n'existe pas. 

6. Appareil selon ia revendication 5 comprenant en 
outre : 

20 

un moyen (49) pour former une image a plus 
haute resolution predite par interpolation des 
valeurs de pixels de la composante d'image 
ajustee a basse resolution ; 
un moyen (50) pour determiner les valeurs de 25 
difference entre les valeurs de pixels interpo- 
ses dans I'image predite a plus haute resolu- 
tion et les valeurs de pixels correspondantes 
dans I'image a plus haute resolution ; 
un moyen (60) pour comparer chacune des va- 30 
leurs de difference aux valeurs de conversion 
de quantificateur pour determiner si une erreur 
existe pour une conversion executee ; 
un moyen (58) pour ajuster les valeurs de pixels 
d'image dans la composante d'image ajustee a 35 
basse resolution si une erreur existe, a une va- 
leur qui elimine I'erreur afin de former une com- 
posante d'image a basse resolution ajustee 
deux fois ; et 

un moyen (56) pour utiliser la valeur de pixel 40 
d'image comme partie de la composante d'ima- 
ge a basse resolution ajustee deux fois si une 
erreur n'existe pas. 

7. Appareil selon la revendication 6, comprenant en *s 
outre : 

un moyen pour appliquer des valeurs de 
pixels aux positions de pixels dans I'image a basse 
resolution pour former une composante d'image a 
basse resolution. 50 

8. Un dispositif de memorisation (10) possedant, m6- 
morisees a Pinterieur, les composantes d'image 
ajustee a basse resolution et les valeurs de pixels 
d'image residuelles de Tune quelconque des reven- ss 
dications 5 a 7. 
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